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ABSTRACT - % . ^ C 

The concepts "fac^t analysi^i" "facet design," and 
"facet structure" 'are? defined. The FYCSP (First Year Communication 
Skills Program) 'Word Attack Test is analyzed in terms of two related 
facet structures. Stepwise linear regression is used to predict \ 
distractor attradtiveness Hypotheses sxiggested . by Guttman relating 
distractor attractiveness to "d^grae of similarity to the correct 
answer". are formulated and tested. Potential applications 6f facet 
analysis aife discussed ^^^^(Authof/DEP) . ^ * 



/ 



r 



* 

* 



****** **|»e3f^^********** 

* I>•pc^J]^e^^^"^5^c'q^^^z^4^r.^^ ERIC dn^l^^^ many informal unpublished ' 
mat•6ri^^^^s.':^;o1^^;^ ^very ^ffort 

Xo obtain :t^b^. >esf,'.c'6p^^^^ item;^; of , marginal -J' 

* reproducib^iii;t'y ' ar4;pfVe affects/ the quality 



* ****t,***^**:***3fc^ *****.*** ********«**3ic* 



AN INI*riAl/>EACET ANALYSIS OF TH^ 
- FYGSP-WORD.-ATtACK-TESJ ^ 



Facet analysis is a collection of ptocedufes tor .^i^cribing the 

content of raultiple-choice'test items and interpreting t\^ observed^tetn 

response patterns. The content of a test item is assumed to 'have two 

asp^^ts: the stimulus and the response options. For the FYCSP word 

att4dlc test, the stimulus is a spoken word which* can be constructed from 

word elements using one or more phonetic rules. For a reading coraprehen- 

sion test, the stimulus may be a selected reading text • (Schlesinger and 

Weiser, 1970). A facet is defined to be a characteristic on which the 

stimulus and an option can be evaluated aijd compared. ^ 

' , 'Facet analysis involves identifying charact?feristics (facets) and 

hypothesizing which are significant in explainir^g respoi:v3e patterns. Re- 

sponse patterns can ^e described quantitatively several ways. A similarity 

measure, s^, for each facet is defined: ^ 

1 if the option and the stimulus are 
identical with respect to facet (i) 

0 if the option and the. stimulus are 
different , 

If more than one facet is used?. to describe a distractor a similarity 



1 



v^ctop/is defined: 



(1) 



The proportion of respondents^^that selected a distradtor is referred to 

■ ' ' ■ . N' ^ • 

as the distractor attract iveness.V ' 



Data Source ' . ^ 

■ ) ^ 

The data available for analysis were the FYCSP criterion exci»ci$es 
from a sample of 10 class-rooms selected from the 1970-71 Quality Assur/ince 
project. Data from 362 students were available from the firs^t two iiViits 
with the sample size shrinkit^ig to 98 s^tudents by' Unit 10/ 'The initial 
use of this data was for developing procedures to. handle efror conditions 
in IMI§ input data (Hooper and McManus, 19-71), To* meet the needs of that 

study, criterion exercise data were scored using procedures simulating 

• - • 

the output of optical scanner interpretation of student responses,, Thiis, 
some responses were scored as multiple-marks in instances wh'ere a human 
could easily distinguish that the student meant to cross-off or eirase 
one response. However, this was not viewed as consequer^ti^l for the 
purposes of this preliminary anal^'sis. ^ 

The sample of test items consists of the 50 word attack items (five- 
per-unit) of the 1970-71 FYCSP criterion exercises. Ea^h item had three , 
response options resulting ifa a total sample size of 100 distractors*. 

' ' , ' ' • ' / ' > 

Possible Facet Structures ' * 

- y 

All of the stimuli for the word attack tests are composed of three 
grapheme* units--an initial consonant ^ sound , a vowel soi^nd and a final 
consonant sound. In word attack instruction, However, eacJi' word is 
treated as an initial consonant sound and a-final vow^l- consonant sound. 
Thus, both a thi;ee facet (C-V-C) and a two facet (C-VC) structure appear 
plausible, • / 



The three facet sttucture results Jn seven classes of di-stractors , 

all of which are represented in the sample of lOCf distractors hut with 

/ 

a very uneven distribution (see Table 1). The three classes ''o'f distrac- 
tors tor the two facet structure are foirtned by collapsing the three facet 
classes: ' * 

^ 

(0,1,1) => v(o.i) 



(1,1,0) 
(1,0,1) 
(1,0,0) 



(0,1,0) 
(0,0,1) 
(0,0,0) 



= > 



=> 



(1,0) 



, (0,0) 

t> 

t 

For example, if the word "saf'is sounded out by the teacher a? a stimulus. 



possible distractors would be classified /as follows: 




Two Facelf' • 


Three Facet 


yStruct«re 


Structure 




' (0.1,1) . 


/ (1.0) s, , 


(i;0,l) 


>(0,0) 


■ (0,0,1) 


. |(0,0) 


(0,0,0) 


, (1,0) 


(1.0,0) 


. (0,0) 


(0.i,0) 


(jl,0) 


(1,1,0) • 



Guttman*9 Hypothesis 

— ' •■ 

GutJ^ppan has suggesteiS the.jfqllowing hypot^^^ 



"...the degree .of attraction of :'a dlstractor increase 
tnonotonely with its, 'degree, ojE similarity ' to €he correct: answer" 
(Guttman and Schlesin^^r^, 1,967).. . ^ • * 



Let A(S) represent the hypothesized relationship between degree of 

similarity and attractiveness, the following partial orders are then 

predicted: . ^ ^ ,^ , ' ■' 

(a(1, 0,0)1 ^ • 

^^<1'1'0> > |a(0,1,0)J \ 

A(l.O.l) > ' jA.(l,0-,0)l ^ A(0,0,0) (3) 
\A-(0,0,1)J 

A(0,1,1) •> JA(0,1,0). > . - A(0,0.0) (4) 



^ fA(0, 1,0)1 ^ 
•\A(0,0,1)J 



Guttman actually proposed a somewhat stronger hypothesis thSn these 
partial orders. Define the level (L) of a distractor to be* the number 
of s values for an item equal to one, 

L = s^ + s^ ^ . . . + s^ . . (5) 

then the attractiv^ess of a Mistractor is predicted to increase mono- 
tonely with respect to level, ^ ^ ' * ^ 

Predicted .Effects of Other Variables 

^ It is expected that other variables , in addition to diistractar- 
Stimulus similarity, may affect attractiveness. It generally is antici- 
-'pated th^t the attractiveness' of jdistractors will decrease afe a student 
progresses thxjough a. program due to learning. . It ^is expect,ed^ th^t this • 
rat^e* of decrease will not fee coh^tant but be greatest for the initial 

v''ijfnits. >, ^ fc, 

It is' expec^d that, if the student doe« not know the correct answer, 
he wiU have a tendency to pick^ the first option. , As a result of learning 
this tendency sfiould be less in evidence ifi later unit tests. ^ 



TABLE 1 - ^ ' ' 

Frequency of Occurrence of Distractor Types as a Function of Unit and Position 



(^hr^erFacet Structure) 
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(1,1, 0)_ 


« 

Ty 

Level 2 
(1,0,1) 


pe of Dis 
(0,1,1) 


tractor 
(1,0,0) 


Level 1 
(0,1,6) 


(0,0,1) 


Level 0 
(0,0,0) 




1 


0 


0' ' 


r 3 
2-0-1 


5 

2-2-1 


1 

0-0-1 


0 • 


1 

0-1-0 






1 

1-0-0 


2 

1-0-1 


4 - 
0-3-1 


2 

0-1-1 


• 1 
, I'-O-O 


0 


0 . 


1 


3 


2 

1-0-1 


1 

0-1-d- 


5 

2-.1-2 


■ 1 
0-1-0, 


0 


1 

0-1-0 


0' 








l-Orl 


0-&2'- 


2 

1-0-1 


1 

0-0-1 


1 

1-0-0 


< 
> 

0 . 




5 


0 


2 

0-1-1 


1-4-0*' 


1 •• 
0-0-1 


■ 0 ;' ■ 


2 

2-0-0 


' . .0 
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. 0 


1 ; 
0-0-1 


5 

1-2-2 •' 


• >v 

0-0-2 


1 

1-0-0 ■ 


1 , 
0-1-0 


0 


7 


1 


2 

2-0-0 


. 5 V 
1-1-3 


2- 

0-0-2 


0 


0 ' 


'0 


8 ' 


2-O7O 


0 . ^ 


4 

1-1-4' 


3 

0-1-2 


. 0 




1 

l_0-0 


• 9' 


3 

1-1-1 


0 


4 

2-1-1 


0-0-i. ^ 


0-0-1 


1 

0-1-0 

2 ^ 


0 

— . 


10 


0-0-1 


1 

1-0-0 


5 

2-2-1 


•2 •• 

0^2-0 , 


1 

1^0-0 


0 




Totals . 


10 
6-1-3 


n 

5-2-4.' 


. 44 
12-17-15 


'21 

3-7-,ll 


6 

3-0-3 


• 6 

3r3-0 


2 

1-1-0 



f «> 



'Notesr Total fb.r a distractor type for a unit is given in the upper' number . 

Positional breakdown, left-center-right, is given beloW if, the cell 
was not empty. ** , • , \ 



Variables are defined in such" a way that their first order effects 



in predicting attractiveness will have preQicted positive Beta weights 
in a linear 'regression equation. ^ 

U = 10 - unit number foV test | containing , the item, 
if-^, I ^ distractop is the fir§t option: 



= ^ 1 K 0 if the distractor is the second option. 
D % 1 . 

1 if the distractor is the. third option. , 

_ t 

The similarity of the two distractors for a item is expected to 
have a secondary effect on the attractiveness of each distractor. !{: 
is hypothesized that inter-distractor similarity reduces the attractive- 
ness of each distractor. Thus, the most attractive distractor is pre- 
dicted to be onjB which is both maximally similar to the correct response 
anU different from its paired distractor. A distractor similarity vari- 
able (DS) is defined to /be equal to the number of facets for which the 
.two distractors are identical, 

Rej^res sion Analys is ^ ' • 5 I/ ' 

Table 1 ^ives the distribution of distractor type by unit and by 
position for the three- facet structure. The column totals indicate that 
the overall occurrence of distractor types is quite unbalanced. Further- 
more, the occurrence of some of the distractor types does not appear to 
be uniform across either unit or position..; Comparison of observed m^an 
attractiveness could, be misleading due to the high frequency. of empty 
cells and a confounding of effects. The hypothesized effects of similarity, 
unit and positiW were^ felt to be best tested by compar^g the predicted 
attractiveness of classes of distractors. 
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Sf^pvls ^ linear re gress ion, was employed to generate 'predictiv e 



equation's of increasing complexity. . As 'displayed in Tables 3 and 4, a, 
linear model with main effects only was first estimated. All of ^he 
Beta weights were positive as predicted. This indicates that'., if onl^^ 
the maiji effects are considered, each of the varialjles effects the 



attractiveness of a distractor in 'the predicted direction. 

Adding the fir^t order interaction ef^e^tjts significantly increases 

the variance accounted for by both regressiouAequations . As predicte'd, ^* 

the Beta weights for the intjeraction of unit' and position C^xPj^) are 

positive. This, is the expected result if the p,osition effect is most 

*' in evidence for the tests early in the program," 

The interaction of a similarity variable, for a facet, with the unit 

, variable is predicted to have a positive Beta weight if it is expected 

-'that the student population ^ill evidence^ reduced tendency to^prck 

' distractofs^, with that similarity, on later unit tests. Thus, if student,s 

learn to"reject every class of distractors, positive Beta, weights for all 

(Uxs^) terms are predicted. If, however, students t^end to learn to reject 

some classes of distractors ai^d not others, then some of the Beta weights 

may be zero or negative. It , is seen that the Beta we^^hts for (Uxs,^)-- 

interaction of unit with initial consonant soufid— is negative. Possible 

implications of this result. will discussed later.. 

Tfie distractor similarity (DS) variable and the second order unit 

• *r 

effect (U ) Were added last to the three' facet model. The variante 

accounted for was significantly increased with most of thet increase^due 

2 - 2 ' ^ ' 

to CU )., The sign of the U Beta weight was positive as predicted. 
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♦ Examination of the type of "distractor similarities occurring revealed 



that there is probably a confounding of distractor similarity with the » 
8^ variables. When' the two distractors in an item exhibited a similarity 
in one or more facets, with but a single exception, they shared that 
sitntlarity with the correct option. If the two distractor^*;had two 

• - • > 

identical facets, they were always the vowel, f ingl-consonah|: gair. 
There was only one occurrence of a distractt)r sitnila|:ity occurring in 
the initial consonant sound. Due to the apparent confounding, interpre- 
tatioiT of the distractor similarity Beta weight was not attempted and 
the second order effects were deleted from the two-facet analysis; 



Discussion of Results * ■ ' 

f Table 2 contains the predicted attractiveness of each class of dis- 
tractors,' for each unit, for the three- facet modfel with the position and, 
distractbr similarity variables set to zero. Figures 1 and 2 compare 
the observed mean attractiveness^of the two xnost frequently occurring 
Classens with the predicted values, <^ 

The partial' order hypotheses of the relative attractiveness of dis- 
tr*actors are supported by the Table 2 values. Of the 90 partial order 
predictions, only sevfen differ from ih^ values predicted by the regression 
equation, * Of the 30 , additional order predictions generated by the "level-*, 
of-distractor" hypotheses, 4 differ from the regression predicti.ons*, * * 

These 4, however, all occurred in the comparison of the two most frequently 

''I . ^ ' - • . ' 

occurring claisses (Figures 1 and 2) and thus constJ.tute stronger evidence 

,^of the weakn^Jis. of, the "level«6f-disttactor" liypotheses, ^ »V 



S) •. • ■ " 

Lie . 



TABLE 2 



Predicted Attractiveness of Distr^ctors and Frequency of their Occut^rence' 
for the FYCSP Word Attack Test. 
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Type? of .Di&tractor 








Unit 
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Level 1 




Level 0- 


X 






(1,1,0) 


(1,0,1) 


(0,1,1) 


•(i-,Q,0) 


(0,1, oy 


(0,0,1) 


(0,0,0) 


• 


• 1 


X 


.118 

(0) 


.141 

(0) 


. 184 

(30 


.068 
' ."(5) 


. 103 ^ 

•• *'r-' ft 


.094 
.(0) . 


.034 
(1) 


\ 

\ 


2 


4' 


-'.09^ 
(1) 

\ > 


.130^. 
(2) ' 


» 150 

(4) 


.056 
"(2) , 


.077 
(1) 


.07-1 
■(0) 


]018 ' 
(0) 
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.077 

' (2) 


.113 

' (1) 


(5) ' 


047 

CD 


.055 
"(0) 


.051 
(1) 


.005 

' (0) 


<> 


4 




.061 
CO') - 


.100 
(2) 


.094.. 
(4). 


.042 ■ 
(2) 


.036 ' 

-(1) 


.035 . 
\ (1). 


-.033 

(0) 




5 


» 


V . 050 
(0) • 


.091 

(2) ■ 


.072 


'.040 
(1) 


^21 
(0) 


.022 

("2) 


-.008 

. (0)' 


/ 


6 




.042 - 
(0) 


.085 
(1) 


.052" 
(5) 


.042 
(2) 


.010 

(1) 


.013 

(1) r 


-.010 

,(0) ' • 
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.03>7 ^ 


.084 
(2) 


.037 
(5) • 


. .04§ ' 
(2) 


.002 
* (0) 


.008 

(0) 
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-.007 
• -(0) 
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.036 \ 
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.085 

(0) 


.024 
(4.) 


.057 
(3) 


-.003- 
XO) 


.006 , 

(0) 


-:do3 
,.a)- 
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.039 

(3) 


.090 

jO) 


•.017 

C4) 


.070 


-.003, 
(1) 


's.OlO 

• (1) 


. 009 . 

(0) 


X 


10 - 




. .046 - 

••' (1) . ■ 


.iDo 
(1) 


.013 

(5)- 


■ .086 • 
• (2) 


.000 

(1;) 


.014 • 

(0)- 


.022 

(0) 
\ — '■ 




Total Number • 




10 


11 


4% 


21 . 

> 


6 


6 


2 




of Distractors 

4 , L_ 









' Notes: (1) The, frequency of occurrence of each type of distractor by unit 
indicated by the numbers in parenthese^s. 

• ' (2)' The Beta weights for the ^full 3-facet ijiodel (third column^of- 

\ , , • ^ ' • ' > 

Table 3) were used with distractor similai^ity^,and position set ^ 
to zero. . ' * 
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^ Figure 1. Attractiveness (A) as a function of progVam unit for distrai l v»n^ will) * 
^P'^ * i' correct initial sound, type (1,0, 0> « ^ 

Solid line - Predict'ed attractiveness, 3-facet full model regression' 
"X ( ) - X)bserved average attractivepess , and sami>lc size 




X|2) 



' X(2) 



X(l) 



X(1) 



. 1 
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2. Attractiveness (A) as a function of program unit for distractors 'with 
correct vowel-consonant final sound, type (0,1,1) 

V Solid line - Predicted attractiveness , L'3-facet full model Degression 
'X ( ) , ' - Obser^^ average attractiveness ^nd^sample size. 




X(5) 



10 ' 



'It' seem^' lively that tKe ^Vleyel^bf-distractox" hypotheses will not 
be suppprted In any ,inst^c'e where one facet, influences attractiveness 
to a subst^anti'ally lesser degree than the o-thex-vfacets-.:. For the word 
^tt^Ck testj the, reiati:ve effects of tb^* '^facets • kegmed fo/ vary' over ; 
time.' • \ . . /T - T-. ' * v- ~:" 

' ^'^'^ ^i' * . \ ' ' ^ - -^-"vy - - 

pifferenrti^l,. learning is. a_*p9tential interpretation of the nqticable* 

differences in slope between th^ regres'sion equations 'for the 'two dis- 

tractpr types plotted in Figures ,1 and 2. If such is the case, inclusion 

• ^ • • ' • * • , 

of mote practice In'^decpding the, medial vowel and final consonant soundJs . 
0& a word may be a desirable program modif icationV^ ;:It is'possible\ how- 
.everf that the observed changes ^n distractor attractiveness across* units 
is due to a tendency t5 speed up test administration "in the^artr units.- 

^ J * ' * 

Facet Design , j . 

* ^* • ■ *t ** * - 

Systematic rules for constructing sets o^ distractors according to • 

an a-priori choice of facets constitutes a facet ^design. Tests based^pn 

fa facet design have the following advantages over, the usual Tirultiple chbic 

te^ts. jjGuttman and Schlesinger, 1967). 



' 1; Succ'es'sfuT predict^n of relative empirica]. ^difficulties of 
distractfors. 

' , : - ■ . " -^ j 

2. Reduction of variation in test results. due" to undesired factors. 
3'. Possibility of differential scoring of subjects* on the, types of 

• . • •'. ' ■ * . '.r . -f' 

' , . wrong answers to which they are attracted. • ' 

, The first two -advantages relate to the potef^ial foy improving the. reli- 
' ability an3rv:^Iidilry^ without increasing test length* The third refer/ 
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to tfie possibility of gaining <fiagnostic informatidn from an ^naly^is of . 
/.the errors a stiident jnakes. . . c \ * * 

A potential* area for both theoretical and empirical research is the " 
determination ofv/which typeis of facet designs are optimal fpr improving 

v; 

test validity.. An unanswered question is whether improving' test validity 
it^creases' or d^creasfes' the amount of diagnostic information that can' be x 

gleaned from (error patterns, ' 

' \r ' ' . * 

Potential Applications of Facet Analysis to IMS , , > 

Facet an,aly?3^.m^y >yield brefiefit^ for each of the; areas curr^ently 

•ittc lauded; in the IMS effort ^,Con^^ of tests according to a 'facet 

desigji would pemutrva systematic invesTt^^gi^tidn of this potential. If the ^ 

student population as a.wtiole exhibits dif fere'ntjUaTr'ifearning in rejecting 

classes of di.stractors , there are implications for progr;^ feVis|.on.. . If 

turthei: analysis reveals that individual students tend to select di-^jrac^' ^ t ; 



%ors ^rpp some class or classes in- a predi^ctable fashion, individualized 



diagnosis and*prescri,ption-may be, indicated. 



^' A f ac 
of an a-pr 



et idesigh «;^eads to the construction of test items tjrre 'basis '>^'< 
iori def init'^on of^^st- confent . *MChis should^b^ a reasonab'l:^:V, .j,;^^ 
starting pbint for dev^^ping computer generated tests and,- computer / ^ > v 
generated drill-an3-practice .exercises* '- ' .^ , " • - T^'-^-y '^^^i-'*'^^^ 



* 

9 



14 



m 



-V. 



H 



TABLE ' ' • 



Linear Keigr^ssibn'Mod^s^f^qr; the --^h'ree Facet 



' 'i. 



''Hi 





/^«^d.n Ef febta 


First. Order 
Interactions 


. Second Order '., 
- ' ECfects.'., ^ 


/ -/ . .•/ /■^•.• 

Co^sp^ant Term ^; v;^ 




.0074 ..• ' 


■." ''"^ y . / ' / . -X C- 

V.,ff21?-' •/' 


/ Main/^ 
Effects 

' ■'/ 


Po s i t ipn //y^^\Ti)': 
SJ.tnilarlty, Fao^t. 1 .-^sf^X' 
, similarity, K^cet 2 ''(s^;);'-^. 
/ Similarity,, Facet 3 
■/Unit; ,^•,1,^ . '-O^V- . (U) 


ioiov. 

/-.C 


- .0029 '. ; :■■ 

- ,059t5. ..•>! 
-.•Q153 

-.0193 '; .^V- 
■■ - ! 0024;'' 


■•■/lx6645^-•■ •.. 


• - /' / -, • 

'4/-: • A-^^i. 

\ p rr - -. qn:t^r a,c jt|^ons 

■ ;•" •• • f :,h'^- 
f : i t'1^ 


;•• ^1 ■4-->.^|iv;- 


\ A' • 

;\ 

x_ ' \ ■ 
x^-\ 


.003*9''. 

. ■- -.;,0039- ; •.- 

... ..-.<- 

- . boBSii,.. ■* 

, •,.0069 

■ ■-!0126r 

.0359 

\ •. 


/A>639^v,' ' • ■.• 
X'^ J.; 003-3; ^ 

.0076 
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^ ' Order. 


^ , U k.s^ . 




♦ • ' ,* * 


* 1 

-.0021' 


Interactions 










[^Multiple Correlation': 




,.592 


«? .748 ^ 














; Variance Accounted 5pr: 


,351 


.559 



V 



ERLC 
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